Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use JSON instead of Yajl to improve performance of store/load files #4759

Merged
merged 1 commit into from
Jan 9, 2025

Conversation

Watson1978
Copy link
Contributor

@Watson1978 Watson1978 commented Jan 6, 2025

Which issue(s) this PR fixes:
Fixes #

What this PR does / why we need it:
When I read 10 GB file using in_tail plugin, it calls Yajl methods many times when it stores to buffer file.

Recently, Ruby's JSON has huge improvement and it is much faster.
Ref. https://byroot.github.io/ruby/json/2024/12/15/optimizing-ruby-json-part-1.html

  • Before

    • It spent 88.533409329 sec to handle 10 GB file
  • After

    • It spent 72.324643557 sec to handle 10 GB file
  • config

<source>
  @type tail
  path /home/watson/prj/sandbox/fluentd/log/access*.log
  pos_file /home/watson/prj/sandbox/fluentd/log/access.log.pos
  tag log
  read_from_head true
  follow_inodes true
  # rotate_wait 0
  refresh_interval 5
  # open_on_every_update true
  <parse>
    @type none
  </parse>
</source>

<match **>
  @type file
  path /home/watson/prj/sandbox/fluentd/log/log
</match>

Docs Changes:

Release Note:

@Watson1978 Watson1978 marked this pull request as ready for review January 7, 2025 00:51
@daipom daipom added this to the v1.19.0 milestone Jan 7, 2025
Copy link
Contributor

@daipom daipom left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants